Roboticized Surgery System with Improved Control

ABSTRACT

A robotized surgery system comprises at least one robot arm which acts under the control of a control console intended for the surgeon. The console comprises an eye tracking system for detecting the direction of the surgeon&#39;s gaze and for entering commands depending on the directions of the gaze detected. The console comprises advantageously a screen with at least one zone for viewing the operating field and, among the commands which can be performed depending on the gaze directions.

The present invention relates to a robotized surgery system, inparticular for mini-invasive operations, such as endoscopies.

In the prior art robotized systems for performing surgical operations,in particular laparoscopic operations, have been proposed. Theserobotized systems comprise one or more robot arms which are controlledby a surgeon via a special console. Said console generally comprises atelevision screen, on which the pictures of the operating field (usuallyrecorded by means of an endoscopic telecamera) are shown, and suitablemanipulators by means of which the surgeon performs the movements of therobot arms. The manipulators may be of the “reactive” type such that thesurgeon is also able to feel with his/her hands the forces exerted bythe robotized surgical instruments on the patient's tissues.

Usually, one of the robot arms moves the telecamera so that the surgeonis able to vary its viewing angle during the operation, while one ormore robot arms move(s) the operating instruments (forceps, scissors,scalpels, aspirators, etc.) with which the surgeon performs the intendedoperation.

It is somewhat complicated for the surgeon to control the system owingto the number of controls available and the number of operatingparameters which the surgeon must oversee. In addition to the movementof the robot arms, the surgeon must also operate the surgical implementsmounted on the arms (for example, control opening and closing of theforceps or scissors) and control the telecamera lens so as to havecloser or more distant pictures of the operating field as required inany one moment.

Often the surgeon, while looking at the picture shown by the telecamera,must simultaneously operate two robot arms with the surgical implements,such that both his/her hands are occupied controlling the correspondingmanipulators.

This prevents the surgeon from being able to use other system functions,unless he/she removes one of his/her hands from the surgical implementcontrols.

Systems have also been proposed where certain functions can be activatedby using pedals or additional control devices provided on themanipulators so that they can be operated with a finger without havingto let go of the manipulator itself. All these systems, however, areoften not intuitive and, in any case, distract the attention of thesurgeon when performing a precise manoeuvre with the surgicalmanipulators.

A further problem is that the surgeon's attention, for one reason oranother (for example, in order to perform further functions or view theoperating parameters shown on the console or on other instrumentsoutside of the operating field), may be distracted from the picture ofthe operating field. In these conditions, the surgeon may not noticeincorrect or dangerous movements of the robotized surgical instruments,which may also be involuntarily performed by him/her while gripping themanipulators.

In order to avoid such situations it has been proposed using a pedalwhich the surgeon must operate in order to activate/deactivate movementof the robots, so as to be able to deactivate the movement beforeremoving his/her hands from the manipulators and directing his/herattention away from the operating field. Timely operation of this pedal,however, depends on the surgeon.

The main object of the present invention is to provide a robotizedsurgery system and a control method which allow better and safer controlby the surgeon, thus ensuring, among other things, better operatingsafety of the system.

In view of this object, the idea which has arisen according to theinvention is to provide a robotized surgery system comprising at leastone robot arm which acts under the control of a control system intendedfor the surgeon, characterized in that the console comprises an eyetracking system for detecting the direction of the surgeon's gaze andfor entering commands depending on the gaze directions detected.

A further idea according to the invention is to provide a method forcontrolling a robotized surgery system comprising at least one robot armwhich acts under the control of a control console intended for thesurgeon, in which the direction of the surgeon's gaze is detected andfunctions of the surgery system are controlled depending on the gazedirection detected.

In order to illustrate more clearly the innovative principles of thepresent invention and its advantages compared to the prior art, anexample of embodiment applying these principles will be described below,with the aid of the accompanying drawings. In the drawings:

FIG. 1 shows a schematic view of a surgery system according to theinvention;

FIG. 2 shows a block diagram of the system according to the invention.

FIGS. 3A and 3B show the steps of a method according to the invention.

FIGS. 4A, 4B and 4C show steps of methods performed using the disclosedsystem.

With reference to the figures, FIG. 1 shows in schematic form arobotized surgery system, denoted generally by 10, provided according tothe invention.

The system 10 comprises at least one robot arm 11 which acts under thecontrol of a control console 12 managed by the surgeon who is forexample sat in a comfortable working position. The console may also bemounted on wheels so that it can be easily displaced.

The robot arm will be of the substantially known type suitable for thespecific use. In the system 10 shown here the robot arms are three innumber, although a different number may be easily used. The robot arm(or each robot arm) terminates in a wrist piece which is able to supportand operate a surgical instrument for use on a patient 13. Thisinstrument will usually be a known instrument for endoscopic, and inparticular laparoscopic, operations. One of the instruments isadvantageously a telecamera 14 which records the operating field (inthis particular case, the patient's interior), while the otherinstruments may be suitable known surgical tools 15, 16 (forceps,aspirators, scalpels, etc.). The robot arms, the instruments and theactuators for manoeuvring these instruments will not be described andillustrated further here, since they are known and may be easilyimagined by the person skilled in the art. The surgical operations whichare possible with the system and the methods for performing them arealso not further described here, since they may be easily imagined bythe person skilled in the art.

The robot arms are operated by a suitable known electronic control unit30 so as to perform the movements entered via the console 12. The unit30 will receive the high-level movement commands (for example, desiredposition and inclination of the tool supported by the robot) and willexecute them, converting them into the corresponding sequences ofsignals to be sent to the individual motors of the robot armarticulations. The robot arms may also be provided with known forcesensors, used both by the unit 30 to prevent damage due to collision ofthe arm with objects in the working space, and in order to providesuitable feedback to the operator at the console, as will be clarifiedbelow. The connections between the console and the unit for controllingthe robots may be advantageously of the optical fibre type, in order toreduce the possibility of interference with the signals transmitted.

Suitable robot arms and control systems are, for example, described inWO2007/088208, WO2008/049898 and WO2007/088206.

In order to perform the movements of the surgical instruments, theconsole may advantageously comprise a pair of known manipulators 17, 18which can be gripped by the surgeon and the movements of which aresuitably reproduced by the surgical instruments by means of movement ofthe robot arms 11. Advantageously, the manipulators may be of the known“reactive” type (i.e. with a tactile interface which provides a weightedmovement and tactile sensing) such that the surgeon is also able to feelon the manipulators the forces exerted by the robotized surgicalinstruments on the patient's tissues. Suitable tactile interfaces arewell-known in the sector of robotized endoscopic surgery.

Usually, each manipulator will operate a robot arm. Advantageously, inthe case of more than two arms, a control will be provided on theconsole so as to be able to assign, as required, each manipulator to adesired robot arm, as will be explained further below. A keyboard 19 andother command input devices, for example also comprising a pedal device20, may also be envisaged. The device 20 may comprise one or more pedalsfor activating, for example, the supply of power for monopolar andbipolar instruments, the irrigation and aspiration functions, ifenvisaged for a specific instrument, etc.

The console 12 also comprises an eye movement tracking system 21 orso-called “eye tracker” for detecting the direction of the surgeon'sgaze towards the console and for controlling the surgical system alsodepending on the gaze directions detected. In this way, the surgeon maycontrol functions of the system by means of movement of the eyes.

Advantageously, the console comprises a video screen 22 with at leastone zone 23 for showing a view of the operating field. This view isprovided by the telecamera 14 which may be supplemented (for example soas to provide the surgeon with additional information and/or improvehis/her understanding of the operating field) with artificial picturesgenerated by a computerized system 24, known per se, for managing theconsole, which will be further described below.

As will become clear below, the computerized system 24 generates andmanages a human machine interface (HMI) which allows the surgeon tointeract with the surgical system.

For conventional two-dimensional viewing, the picture supplied by theendoscope may be directly viewed on the screen 22.

Advantageously, however, a three-dimensional system for viewing theoperating field may be envisaged. In this case, the telecamera 14 may beof a known stereoscopic type which provides suitable signals 25, 26representing two different “right-hand” and “left-hand” pictures whichare recorded spatially offset. The signals 25, 26 are processed by anelectronic device 27 so that the 3D picture may be shown to the surgeonby means of a stereoscopic viewing system.

From among the various known stereoscopic viewing systems a polarizedfilter system has been found to be particularly advantageous; in thissystem the electronic device 27 comprises a known stereo mixer whichalternates lines of the right-hand and left-hand pictures received fromthe telecamera so as to display them interlaced in the viewing area 23of the screen. Alternate odd and even horizontal lines of the picture onthe screen thus represent alternately lines of the right-hand andleft-hand pictures recorded by the telecamera.

A known filter provided with two different polarization modes for theeven interlaced lines and the odd interlaced lines is provided in thearea 23 for viewing this interlaced picture. In order to view thepicture, the surgeon wears glasses 28 with the two lenses polarized in amanner corresponding to the two polarization modes of the filter on thescreen, so as to direct towards the right eye only the lines of thepicture belonging to the original right-hand picture recorded by thetelecamera, while the left-hand eye receives only the lines of thepicture belonging to the original left-hand picture recorded by thetelecamera.

It is thus possible to show the surgeon the desired 3D picture of theoperating field.

If desired, using a similar procedure, artificial stereoscopic picturesproduced by the computerized system 24 may also be shown in 3D form.

In any case, by means of the tracking system 21 it is possible to detectthe direction of the surgeon's gaze towards the screen 22 and definewhich zone of the screen he/she is looking at or not looking at. Using a3D viewing system with polarized glasses there is no interference withthe tracking system. Moreover, glasses with polarized lenses for 3Dviewing can be easily designed so as to be compatible with the normaleyeglasses.

It has been found to be particularly advantageous for the trackingsystem to send a command which disables the movement of the robot armswhen a direction of the gaze which falls outside of the screen, or atleast outside of the screen zone which reproduces the operating field,is detected. See FIG. 4C. In this way, a safety system preventingmovements of the arms without direct supervision of the surgeon isprovided.

A so-called “dead man's” function is thus obtained for activating therobot and keeping it activated while the user is looking at the screen.

Advantageously, for additional safety, a further control means may beprovided (for example a pushbutton 31 on a handgrip or a pedal device20) where it is required to give a dual consent for enabling themovement commands so that, in order to reactivate the movement, thesurgeon must look at the picture on the screen and also give a manualconsent command, while the movement may be interrupted by simply lookingaway from the picture.

Advantageously, the screen 22 shows, in addition to the view from theendoscope, also at least part of the human machine interface. Thecomputerized system 24 which provides the interface shows on a screenselection areas 29 associated with system commands. Advantageously, theselection areas may be arranged on the same screen 22 which shows theview of the operating field. For example, these selection areas may bearranged in the bottom part of the screen, underneath the area 23 forviewing the operating field. The tracking system estimates the directionof the surgeon's gaze and performs selection of the commands associatedwith a selection area when it detects a gaze direction which fallswithin this area.

The commands associated with the various selection areas may be of anytype considered to be useful. For example, these commands may be chosenfrom among those which are frequently used when performing a robotizedsurgery operation.

It has been found to be particularly advantageous (in particular whenthe console comprises two operating manipulators and more than two robotarms) if the commands associated with the selection areas comprise thecommands for assigning the manipulators to the robot arms.

The surgeon may thus alternate control of the various robot arms on thetwo manipulators, without letting go of the manipulators, but insteadsimply looking at the corresponding selection areas. For example, thesurgeon may temporarily switch over to control of the arm with thetelecamera, in order to modify the view of the operating field, and thenrapidly return to control of the robot arm with which he/she wasoperating.

For additional safety, the console may advantageously comprise a devicefor inputting a special command confirming execution of the commandassociated with the selection area looked at. This device mayadvantageously be a pushbutton 31 which is arranged on one or both themanipulators, so as to be pressed, for example, using the thumb of thehand gripping the manipulator. It is thus possible to confirm easily theactions activated by the eyes via the eye tracking system, for examplein order to select a robot to be associated with the manipulator,open/close the surgical instruments and modify the settings of the robotwhich is being operated.

Another use of the pushbutton may also be that of controlling the degreeof freedom of a twisting movement on the instrument (if available).

It is also possible to envisage advantageously that the procedure forassigning a robot may be performed by visually selecting the picture ofthe new robot which is to be assigned, confirming the selection by meansof the pushbutton and then dragging the picture selected into theposition where the picture of the robot currently assigned to theright-hand grip or left-hand grip is shown. Dragging is performed bykeeping the pushbutton pressed and directing one's gaze towards therobot position. In order to end the dragging operation, the pushbuttonmust be released while keeping one's gaze focused on the previouslyindicated zone.

The eye tracking system may be one of the many types which are known perse. However, an eye tracking system which has been found to beparticularly advantageous is one comprising at least one telecamera forrecording the picture of at least the surgeon's eyes and means forcalculating the direction of the gaze depending on the picture taken.

In particular, as shown schematically in FIG. 2, the tracking system 21may comprise two telecameras 32, 33 which are arranged alongside eachother at a suitable distance so as to record two spatially offsetpictures of the surgeon's eyes. The calculation means (for examplecomprising a suitably programmed microprocessor) present in the trackingsystem 21 may thus perform a triangulation of the gaze directiondepending on the comparison of the two recorded pictures. Againadvantageously, the tracking system may also comprise an infrared lightsource 34 for infrared illumination of the eyes, this facilitatingdetection thereof in the picture recorded. Advantageously the eyetracking system may be integrated with the monitor so that if the latteris moved, the eye tracker may continue to operate correctly.

Still with reference to FIG. 2, a block diagram of a possibleadvantageous embodiment of the console is schematically shown. In thisembodiment, the system is divided up for the sake of clarity into threemain functional blocks or groups.

The first block, indicated by 40, comprises the components which areinvolved directly in the movement of the robot arms. The block 40contains a first industrial computer 41, known per se, provided with areal-time operating system (for example, RT-LINUX) for carrying out in agiven predefinable time the commands associated with control of therobots. The computer 41 is connected to the robot control unit (orunits) 30 via the communications network 42. The computer 41 receivesthe movement commands from the manipulators 17 and 18, sending them tothe robots and emitting signals for operation of the reactive devices 43of the manipulators for tactile feedback. Those manual controls whichrequire an immediate system response, such as the pedals 20, if used tosend, among other things, commands for stopping the movement of therobot, are also advantageously connected to the computer 41.

The second functional block, which is indicated by 44, comprises asecond industrial computer 45 which produces and controls the humanmachine interface (HMI) which does not require strictly real-timeoperation. The eye tracking system 21, the keyboard 19 (where necessary)and the other interface controls are connected to this second computer.The computer 45 also produces the artificial video pictures to bereproduced on the screen 22 (for example the visual control areas 31)and may control any functions for varying the enlargement of theoperating field picture.

The computers 41 and 45 form the computerized system 24 for controllingthe console.

The computers 41 and 45 and the robot control unit 30 may communicatewith each via the network 42. The HMI application managed by thecomputer 45 thus allows the robots to be assigned to the manipulators,as well as display of the data relating to each robot, such as theinstruments currently mounted, the movement state, the feedback state,the position of rotational fulcrums of the instruments inserted insidethe patient's body, the robot condition, the robot connection state, anyemergency conditions, etc.

The third functional block, indicated by 46, deals with reproduction ofthe pictures on the screen, providing for example the PiP(Picture-in-Picture) function using the signal supplied by thetelecamera 14 recording the operating field and the picture signal 47produced in order to display the HMI interface. The third block alsocomprises the stereo mixer 27 for three-dimensional viewing.

Advantageously, for the PiP function the monitor 22 is designed with twoseparate inputs. The main source is displayed in full screen mode bymeans, for example, of a DVI connection, while at the same time anothervideo input (for example a VGA connection) is displayed as an insetwindow. The main source (full screen) consists of the 2-dimensional or3-dimensional view of the endoscope which is received from the endoscopesystem. The second source comes from the computer 45 which produces thehuman machine interface (HMI).

During calibration of the eye tracking system 21, the full-screen viewmay also be switched dynamically (for example, by means of a serialcommand sent from the HMI application to the monitor) to the videosignal produced by the computer 45.

Advantageously, the console may also comprise a system for detecting thedistance between screen and surgeon's eyes in order to vary enlargementof the picture of the operating field shown on the screen depending on avariation in the distance detected.

Thus, the surgeon may intuitively perform enlargement of the picture bysimply moving his/her face towards the screen and, vice versa, increasethe viewing area of the operating field, thus reducing enlargement, bymoving his/her face away from the screen.

The distance detection system may be achieved in various ways, known perse, for example using telemetric ultrasound measurement devices.

Advantageously, however, the eye tracking system 21 may be used, owingto the stereoscopic recording system which allows calculation, by meansof triangulation, of the distance of the surgeon's face. This, togetherwith the associated eye detection function of the eye tracking system,allows an accurate real measurement of the distance of the surgeon'sviewpoint from the screen to be performed.

In addition to an enlargement, the system may also produce adisplacement of picture, for example so as to centre the picture, ordisplace the picture, by means of the gaze, to the right, to the left,upwards or downwards.

When the operating field is recorded by a telecamera, preferably anendoscopic telecamera, mounted on one of the robot arms, it has beenfound to be advantageous for the gaze detection tracking system to allowalso control of the movement of this telecamera in the manner depictedin FIGS. 4A, 3A and 3B. When this function is enabled (for example, byentering, by means of visual selection of a suitable area 29, theassociated activation command), the movement of the eyes over thepicture of the operating field causes the movement of the robot arm soas to displace, and advantageously centre on the screen, the zonefocused on. Control of the actual movement of the telecamera may also beperformed only following pressing of a confirmation pedal or pushbutton,as already described above. In this way, the surgeon is free to movehis/her eyes over the picture without displacement of the viewing frameunless the confirmation pedal or pushbutton is simultaneously pressed.If the movement and enlargement functions described above are combined,the system becomes very easy to control, the movements of the gazedisplacing the viewing frame on the screen, while the movement of theeyes towards or away from the screen enlarges or diminishes the pictureshown.

When three-dimensional viewing of the operating field is used, thedistance detection system may also be used to signal to the surgeon whenhe/she is within the optimum distance range from the screen.

In fact, usually three-dimensional systems have an optimum distanceinterval from the screen where the three-dimensional effect is best.

Moreover, the combination of 3D and eye tracking system imposes certainconstraints with regard to the position and the distance from thescreen, said constraints depending on the position of the surgeon, thetracker and the viewing device.

The HMI application of the console may be set so as to indicate to thesurgeon, by means of various known acoustic and/or optical systems, whenhe/she is situated in the optimum position with respect to the screen.Moreover, it is also possible to provide a function which indicateswhether the distance with respect to the eye tracker is adequate. Whenthe 3D and eye tracker are used together, the appropriate working spacemay be the same for both of them and the same indicator may perform bothfunctions.

The working area of the eye tracker 21 will generally be chosen so as tobe much greater than that for optimum viewing of the three-dimensionalpicture. For example, an operating range of the tracker lying between 40and 75 cm has been found to be advantageous, with the possibility oftracking the eyes within a vertical angle of +30° and −10°. The optimum3D view is obtained at 60-70 cm from the screen (in this range, theinformation will be perfectly differentiated between right eye and lefteye) and therefore falls well within the operating zone of the tracker.However, it will still be possible to view in 3D outside of this range,provided that the surgeon respects the vertical constraint. Beyond thetop end and bottom end of the screen, the 3D is lost.

At this point it is clear how the predefined objects have been achieved.With the control system and method described it is possible to controlthe robot arms with tactile sensing, display the view provided by theendoscope in two or three dimensions, together with the HMI application,and activate certain functions using an eye movement tracker. Owing tothe use of the eye tracking system, various interesting controlpossibilities exist. Firstly, among the various functions associatedwith the eye tracking system, there is that of being able to stop safelythe movement of the robot arms if the surgeon is not looking at thepicture of the operating field, with movement of the robot arm which isprevented or allowed automatically when the gaze direction detected doesnot fall or falls within the predetermined zone of the screen.

Moreover, the HMI application is intuitive and easy to use since it maybe controlled by the surgeon's gaze (together with or without anactivation confirmation device). The main advantages are that thesurgeon is able to use his/her eyes in order to select and assign therobot arms to the manipulators without removing his/her hands from themanipulators.

Obviously, the above description of an embodiment applying theinnovative principles of the present invention is provided by way ofexample of these innovative principles and must therefore not beregarded as limiting the scope of the rights claimed herein. Forexample, the console forms a remote operation surgical workstation forthe robotized system which may be in the same room or at a distance,also using a connection via geographic networks or the like. The maincontrol console is in fact a remote control device which also allows apatient to be operated on outside of the operating theatre and in anylocation, as long as the communications time delays are limited. Theremote surgery system will be suitable for any type of laparoscopic orsimilar operation. Obviously, here the term “surgeon” is understood asmeaning any person who controls the robot system by means of theconsole.

It must be noted that, as can now be easily imagined by the personskilled in the art, the system according to the invention is modular andmay for example be configured to use a greater number of robots (forexample up to five robots) and also one or two control consoles.

We claim:
 1. A surgical system comprising: a first robotic arm carryinga surgical instrument; a second robotic arm carrying an endoscopiccamera, a surgeon console including a display displaying an image fromthe endoscopic camera; an eye tracking system, and a manipulatormoveable by a user's hands to generate movement command signals for thefirst robotic arm, the manipulator further including amanually-actuatable auxiliary input; the system operable in an eyetracking mode in which movement of the second robotic arm is commandedbased on signals from the eye tracking system corresponding to anestimated direction of the user's gaze towards the image, the systemconfigured to enter into eye tracking mode in response to user actuationof the auxiliary input on the manipulator.
 2. The system of claim 1,wherein the first and second robotic arms are deactivated in response tosignals from the eye tracking system indicating that the estimateddirection of the user's gaze is outside an area of the display.
 3. Thesystem of claim 1, wherein the system is configured such that movementof the manipulator during activation of the auxiliary input results inmovement of the first robotic arm and corresponding movement of thesurgical instrument, and simultaneous movement of the second robotic armto reposition the instrument in response to input from the eye trackingsystem.
 4. The system of claim 1, wherein the auxiliary input is on ahand grip of the manipulator.
 5. The system of claim 4, wherein theauxiliary input is a button.
 6. The system of claim 1, wherein theauxiliary input is a button.
 7. A method of using a surgical roboticsystem, comprising: providing a surgical system comprising a firstrobotic arm carrying a surgical instrument; a second robotic armcarrying an endoscopic camera, a surgeon console including a displaydisplaying an image from the endoscopic camera; an eye tracking system,and a manipulator moveable by a user's hands, the manipulator furtherincluding a manually-actuatable auxiliary input; causing movement of thefirst robotic arm in response to movement of the manipulator, causingthe system to enter an eye tracking mode in response to actuation of theauxiliary input on the manipulator, wherein, in the eye tracking mode,movement of the second robotic arm is commanded based on signals fromthe eye tracking system corresponding to an estimated direction of theuser's gaze towards the image.
 8. The method of claim 7, furtherincluding deactivating the first and second robotic arms in response tosignals from the eye tracking system indicating that the estimateddirection of the user's gaze is outside an area of the display.
 9. Themethod of claim 7, wherein movement of the manipulator during activationof the auxiliary input results in movement of the first robotic arm andcorresponding movement of the surgical instrument, and simultaneousmovement of the second robotic arm to reposition the instrument inresponse to input from the eye tracking system.
 10. The method of claim7, wherein actuation of the auxiliary input comprises actuating an inputon a hand grip of the manipulator.
 11. The method of claim 10, whereinactuation of the auxiliary input comprises pressing a button on the handgrip.
 12. The method of claim 7, wherein actuation of the auxiliaryinput comprises pressing a button on the hand grip.